Skip to content

Complete the human-grounded promotion evidence loop - #3

Merged
jon-devlapaz merged 6 commits into
mainfrom
codex/business-ready-promotion-review
Aug 17, 2026
Merged

Complete the human-grounded promotion evidence loop#3
jon-devlapaz merged 6 commits into
mainfrom
codex/business-ready-promotion-review

Conversation

@jon-devlapaz

Copy link
Copy Markdown
Owner

Outcome

Turns a repeated promotion run into an auditable evidence package that a human decision owner can review.

What changed

  • adds prepare-review to generate a blinded, hash-bound A/B review packet from a valid promotion run
  • adds finalize-review to validate an independent holdout attestation and exactly two complete human reviews
  • reports human agreement, automated-to-human agreement, disagreements with rationales, per-task outcomes, trial variance, regressions, usage, and operator-recorded cost
  • retains and hash-lists the manifest, holdout attestation, and both reviewer label files
  • documents the holdout custodian, evaluator operator, reviewers, and decision owner handoff
  • keeps the automated OpenAI judge explicitly provisional and non-independent
  • leaves the promotion decision with an accountable human owner

Scientific boundary

This PR completes the framework workflow. It does not invent a client holdout, fabricate human labels, or claim that any target skill passed promotion. A client-specific promotion result still requires real independently controlled tasks and two real reviewers.

Verification

  • 50 unit and end-to-end tests pass
  • Ruff passes
  • Python compilation passes
  • packaged healthcheck passes
  • standalone copied-skill healthcheck passes
  • diff check passes

@jon-devlapaz
jon-devlapaz marked this pull request as ready for review August 17, 2026 00:08
@jon-devlapaz
jon-devlapaz merged commit ed737d9 into main Aug 17, 2026
12 checks passed
@jon-devlapaz
jon-devlapaz deleted the codex/business-ready-promotion-review branch August 17, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant